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Estimate  Costs  of  Reports  and  Studies 

Citing  long-term  fiscal  challenges  affecting  the  federal  government,  in  May  2010,  the 
Secretary  of  Defense  directed  the  Department  of  Defense  (DOD)  to  undertake  a 
departmentwide  initiative  to  assess  how  the  department  is  staffed,  organized,  and 
operated  with  the  goal  of  reducing  excess  overhead  costs  and  reinvesting  these 
savings  in  sustaining  DOD’s  current  force  structure  and  modernizing  its  weapons 
portfolio.1  The  Secretary’s  initiative  targeted  both  shorter-  and  longer-term 
improvements  and  set  specific  goals  and  targets  for  achieving  cost  savings  and 
efficiencies.  The  initiative  was  organized  along  four  tracks,  each  of  which  had  a 
different  focus  (see  enc.  I).  The  fourth  track  focused  on  specific  areas  where  DOD 
could  take  immediate  action  to  reduce  inefficiencies  and  overhead,  in  particular,  to 
reduce  headquarters  and  support  bureaucracies  and  to  instill  a  culture  of  cost 
consciousness  and  restraint  in  the  department.  As  part  of  the  fourth  track,  the 
Secretary  of  Defense  announced  a  number  of  specific  initiatives,  including  actions 
intended  to  address  the  need  to  reduce  or  eliminate  reporting  requirements  for  DOD 
reports  and  studies.  For  example,  in  his  August  9,  2010,  speech  announcing  the 
overall  efficiency  initiative,  the  Secretary  of  Defense  stated  that  the  department  is 
“awash  in  taskings  for  reports  and  studies”  and  directed  several  specific  actions  that, 
according  to  the  press  release  accompanying  the  announcement,  were  intended  to 


1  Remarks  as  delivered  by  former  Secretary  of  Defense  Robert  M.  Gates,  Abilene,  Kansas,  May  8,  2010. 
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“combat  the  enormous  amounts  of  taskings  for  reports  and  studies.”2  In  his  remarks, 
the  Secretary  noted  that  there  is  little  basis  to  determine  whether  the  value  gained  is 
worth  the  considerable  time  and  resources  expended  to  generate  reports  and 
studies.  With  respect  to  specific  actions,  the  Secretary  directed  DOD  and  its 
components  to  track  the  approximate  cost  of  preparing  DOD  reports  and  studies  and 
publish  the  cost  on  the  front  cover  of  each  report  or  study.  He  also  called  for  a 
comprehensive  review  of  all  oversight  reports  and  use  of  the  results  to  reduce  the 
volume  of  reports  and  studies  generated  internally  while  engaging  with  Congress  on 
ways  to  meet  their  needs  while  working  together  to  reduce  the  number  of  reports. 


In  September  2010,  the  Secretary  of  Defense  tasked  DOD’s  Cost  Assessment  and 
Program  Evaluation  (CAPE)  office  with  implementing  the  Secretary’s  requirement  to 
publish  the  cost  of  preparing  DOD  reports  and  studies.  By  November  2010,  the 
CAPE  office  had  developed  a  cost  estimating  tool  and  corresponding  guidance  on 
using  the  tool,  which  described  the  categories  of  costs  to  be  estimated  and  identified 
the  types  of  reports  and  studies  for  which  costs  were  to  be  estimated.3  The  following 
month,  the  Secretary  of  Defense  issued  a  memorandum  requiring  the  use  of  the  tool 
beginning  on  February  1 , 201 1 ,4 


In  House  Report  Number  1 12-78,  which  accompanied  a  bill  for  the  National  Defense 
Authorization  Act  for  Fiscal  Year  201 2, 5  the  House  Committee  on  Armed  Services 
commended  DOD  for  implementing  a  process  for  collecting  an  estimate  of  resources 
required  to  generate  internally  and  externally  required  reports  and  agreed  that 
additional  transparency  would  be  useful  for  decision  makers  when  determining  the 
utility  of  various  reporting  requirements.  The  committee  also  observed  that  any  tool 
used  to  collect  costs  was  only  as  useful  as  the  inputs  received  and  directed  GAO  to 
conduct  an  assessment  of  DOD’s  methodology  and  tools  for  collecting  cost  data  on 
both  internally  and  externally  required  reports  and  to  submit  a  report,  including  any 
recommendations  needed  to  improve  the  data  collection,  transparency,  and  utility  of 
the  tool.  Specifically,  we  evaluated  DOD’s  approach  to  estimate  the  costs  of 
selected  reports  and  studies,  including  (1)  whether  DOD  entities  are  capturing  and 
presenting  costs  in  a  manner  consistent  with  relevant  cost  estimating  best  practices; 
(2)  the  status  of  DOD’s  efforts  to  track  whether  organizations  are  developing  cost 
estimates  for  all  required  types  of  reports  and  studies;  and  (3)  whether  DOD  has 
evaluated  the  usefulness  of  its  efforts  to  estimate  the  costs  of  reports  and  studies  as 
one  of  the  means  for  achieving  the  Secretary’s  intended  purpose  of  increasing  the 
transparency  of  costs,  reducing  or  eliminating  reporting  requirements,  and  instilling  a 
culture  of  cost  consciousness  across  DOD. 


2  Remarks  as  delivered  by  former  Secretary  of  Defense  Robert  M.  Gates,  The  Pentagon,  August  9,  2010,  and 
DOD  News  Release  No.  706-10,  “Sec.  Gates  Announces  Efficiency  Initiatives”  (Aug.  9,  2010). 

3  Guidance  for  using  the  tool  has  been  updated  and  the  most  recent  version  can  be  found  in  DOD  Cost 
Guidance  Group,  Studies  and  Reports  Guidance  Document:  DOD  Cost  Guidance  Portal  (Sept.  16,  201 1 ),  and 
DOD  Cost  Guidance  Group,  Cost  Guidance  Frequently  Asked  Questions:  DOD  Cost  Guidance  Portal  (Sept.  16, 
2011). 

4  Secretary  of  Defense  Memorandum,  Consideration  of  Costs  in  DOD  Decision-Making  (Dec.  27,  201 0). 

5  H.R.  Rep.  No. 1 12-78  (2011),  which  accompanied  a  bill  for  the  National  Defense  Authorization  Act  for  Fiscal 
Year  2012,  Pub.  L.  No.  112-81  (2012). 
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To  determine  whether  the  tool  is  being  used  to  capture  and  present  costs  in  a 
manner  consistent  with  relevant  cost  estimating  best  practices,  we  reviewed  the  cost 
estimating  tool  and  compared  it  to  characteristics  of  cost  estimating  best  practices 
identified  in  GAO’s  Cost  Estimating  and  Assessment  Guide 6  and  related  managerial 
cost  accounting  guidance.7  GAO’s  Cost  Estimating  and  Assessment  Guide  reflects 
best  practices  for  developing  and  managing  cost  estimates  for  capital  programs. 
Because  DOD  is  estimating  noncapital  costs,  we  assessed  DOD’s  tool  used  to 
generate  cost  estimates  against  the  best  practices  most  applicable  to  noncapital 
costs  as  well  as  against  relevant  managerial  cost  accounting  standards  and 
guidance,  which  we  determined  are  consistent  with  the  selected  best  practices. 
Specifically,  we  determined  the  extent  to  which  the  estimates  resulting  from  use  of 
the  tool  were  “comprehensive,”  “well  documented,”  and  “accurate” — essential 
elements  for  high-quality,  reliable  cost  estimates.  We  also  interviewed  DOD  officials 
to  corroborate  our  understanding  of  their  use  of  the  tool  and  its  corresponding 
guidance.  All  data  elements  we  assessed  are  sufficiently  reliable  for  the  purpose  of 
assessing  DOD’s  cost  estimating  tool  against  best  practices.  We  had  originally 
intended  to  review  a  representative  sample  of  DOD  reports  and  studies  for  which 
cost  estimates  had  been  generated  and  published.  However,  as  described  in  this 
report,  DOD  does  not  currently  have  the  ability  to  distinguish  between  real  estimates 
and  estimates  that  were  developed  while  individuals  were  using  the  tool  for  training 
purposes.  As  a  result,  DOD  could  not  provide  us  with  a  comprehensive  list  of  all 
reports  and  studies  for  which  costs  had  been  estimated.  However,  DOD  officials 
were  able  to  provide  us  information  on  nine  reports  prepared  for  Congress  for  which 
cost  estimates  had  been  generated  in  201 1 .  We  reviewed  these  nine  reports  and 
interviewed  the  key  officials  responsible  for  developing  the  cost  estimates  for  them. 
As  described  in  this  report,  in  some  cases,  officials  were  not  able  to  readily  retrieve 
documentation;  therefore,  we  were  unable  to  fully  evaluate  the  extent  to  which  they 
consistently  applied  DOD’s  guidance  for  estimating  costs.  To  describe  the  status  of 
DOD’s  efforts  to  track  whether  organizations  are  developing  cost  estimates  for  all 
required  types  of  reports  and  studies,  we  reviewed  guidance  and  interviewed  DOD 
officials  to  assess  the  nature  of  their  tracking  efforts.  To  determine  whether  DOD  has 
evaluated  the  usefulness  of  its  efforts  to  estimate  the  costs  of  reports  and  studies  as 
one  of  the  means  for  achieving  the  Secretary’s  intended  purpose  of  increasing  the 
transparency  of  costs,  reducing  or  eliminating  reporting  requirements,  and  instilling  a 
culture  of  cost  consciousness  across  DOD,  we  reviewed  DOD  guidance.  We  also 
interviewed  DOD  officials  responsible  for  developing  and  using  the  tool,  including  the 
individuals  who  generated  the  nine  reports  that  we  reviewed  to  determine  any  steps 


6  GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing  Capital 
Program  Costs,  GAO-09-3SP  (Washington,  D.C:  March  2009).  GAO  developed  this  guide  in  order  to  establish  a 
consistent  methodology  that  can  be  used  across  the  federal  government  for  developing,  managing,  and 
evaluating  cost  estimates. 

7  Federal  Accounting  Standards  Advisory  Board,  Statement  of  Federal  Financial  Accounting  Standards  (SFFAS) 
No.  4,  Managerial  Cost  Accounting  Concepts  and  Standards  for  the  Federal  Government  (Washington,  D.C.: 
July  1995),  and  DOD  Financial  Management  Regulation  7000. 14R,  Accounting  Policy  and  Procedures,  vol. 
4  (2012),  see  ch.  19,  “Managerial  Cost  Accounting,”  and  ch.  22,  “Cost  Finding.” 
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taken  to  evaluate  the  impact  of  the  cost  estimating  effort,  including  whether  they  had 
obtained  feedback  from  internal  and  external  decision  makers. 


We  conducted  this  performance  audit  between  October  201 1  and  May  2012  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 
evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions  based  on 
our  audit  objectives.  We  believe  that  the  evidence  obtained  provides  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objectives.  More  detailed 
information  on  our  scope  and  methodology  is  provided  in  enclosure  II. 


Results  in  Brief 


DOD  is  estimating  and  publishing  approximate  costs  for  selected  types  of  internally 
and  externally  required  reports,  but  in  some  cases  its  approach  is  not  fully  consistent 
with  relevant  cost  estimating  best  practices  and  cost  accounting  standards. 
Specifically,  DOD  entities  have  been  directed  to  use  the  cost  estimating  tool  to 
capture  marginal  costs  of  activities  associated  with  completing  a  report  or  study  that 
would  not  have  been  performed  otherwise.  These  costs  consist  of  certain  manpower 
costs  (such  as  the  prorated  salaries  of  military  and  civilian  personnel  based  on  the 
time  they  spent)  and  nonlabor  costs  (such  as  contract  services,  travel,  or  printing).  In 
comparing  DOD’s  approach  to  (1)  GAO’s  Cost  Estimating  and  Assessment  Guide — 
which  states  that  high-quality,  reliable  cost  estimates  should  be  comprehensive,  well 
documented,  and  accurate — and  (2)  relevant  accounting  standards,  we  found  the 
following. 


Comprehensive.  GAO’s  cost  guide  states  that  cost  estimates  should  include  all 
costs,  but  allows  flexibility  for  the  estimates  to  exclude  costs  where  information  is 
limited  as  long  as  steps  are  taken  to  clearly  define  and  document  what  costs  are 
included  or  excluded.  DOD’s  guidance  on  using  the  tool  defines  and  documents 
decisions  to  include  certain  manpower  and  all  nonlabor  costs  in  its  calculation.  In 
addition,  it  documents  the  decision  to  exclude  other  manpower  costs,  such  as  those 
for  health  care  and  training  expenses.  It  also  provides  broad  examples  of  some 
types  of  activities  to  consider  in  estimating  costs,  but  leaves  it  up  to  the  discretion  of 
the  individuals  generating  the  cost  estimate  to  decide  which  types  of  activities  to 
include  or  exclude.  As  a  result,  based  on  the  nine  reports  we  reviewed,  we  found 
inconsistencies  in  the  types  of  activities  individuals  decided  to  include  when 
developing  their  cost  estimates.  For  example,  we  found  that  for  six  of  the  nine 
reports  we  reviewed,  some  individuals  calculated  the  manpower  costs  for  activities 
associated  with  coordinating  the  report  through  the  final  review  while  others  did  not. 
In  addition,  according  to  relevant  accounting  standards,  appropriate  procedures  and 
practices  should  be  established  to  enable,  among  other  things,  the  interpretation  and 
communication  of  cost  information.  However,  when  presenting  the  cost  estimate  on 
the  front  cover  of  a  report  or  study,  the  language  DOD  uses  does  not  provide 
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information  on  what  costs  are  included  or  excluded.  Specifically,  it  does  not  indicate 
that  the  estimate  reflects  only  certain  marginal  manpower  and  nonlabor  costs. 
Without  further  explanation,  this  wording  could  be  subject  to  misinterpretation  such 
that  recipients  of  the  reports  or  studies  may  assume  that  all  costs  are  included  in  the 
estimate  when  that  is  not  the  case. 

Well  documented.  GAO’s  Cost  Estimating  and  Assessment  Guide  states  that  cost 
estimates  should  be  easily  traceable  to  source  documents.  While  DOD’s  guidance 
on  using  the  tool  states  that  individuals  should  be  prepared  to  explain  how  they 
developed  the  cost  estimate,  it  does  not  include  a  requirement  for  individuals  to 
retain  source  documentation.  In  practice,  we  found  that  of  the  nine  reports  we 
reviewed,  individuals  were  able  to  easily  retrieve  documentation  for  the  estimates 
prepared  for  three  reports,  but  not  for  the  other  six.  As  a  result,  DOD  may  not  have 
the  information  needed  so  that  others  can  understand  how  a  cost  estimate  was 
derived  and  readily  replicate  it. 


Accurate.  GAO’s  Cost  Estimating  and  Assessment  Guide  further  states  that  cost 
estimates  should  have  all  cost  inputs  checked  to  verify  that  they  are  accurate. 
However,  we  found  that  DOD’s  guidance  on  using  the  tool  has  no  requirement  to 
independently  verify  the  costs  used  to  generate  the  cost  estimate.  In  practice,  the 
individual  generating  the  cost  estimate  is  responsible  for  ensuring  its  accuracy.  As  a 
result,  without  independent  verification,  DOD  cannot  ensure  that  cost  estimates  are 
accurate  and  reliable. 


DOD  currently  lacks  a  means  to  ensure  that  organizations  are  developing  cost 
estimates  for  all  required  types  of  reports  and  studies,  but  is  taking  steps  to  enhance 
its  ability  to  monitor  the  preparation  of  reports  and  studies  to  satisfy  reporting 
requirements,  including  those  for  which  a  cost  estimate  is  required  to  be  generated. 
DOD’s  guidance  on  using  the  tool  identified  10  specific  types  of  reports  or  studies 
that  require  a  cost  estimate.  The  guidance  states  that  the  DOD  component  preparing 
a  report  or  study  is  responsible  for  ensuring  that  a  cost  estimate,  if  required,  is 
included.  However,  the  guidance  does  not  include  any  process  or  requirement  to 
track  whether  organizations  are  developing  required  cost  estimates.  According  to 
CAPE  officials  who  developed  the  tool,  they  were  not  directed  to  ensure  that  cost 
estimates  have  been  developed  for  all  required  reports  and  studies,  and  they  added 
that  the  cost  estimating  tool  was  not  designed  to  track  reports  and  studies  that 
require  a  cost  estimate.  These  same  officials  noted  that  the  magnitude  of  DOD’s 
reporting  requirements  makes  it  challenging  to  identify  the  universe  of  these 
requirements  and  to  track  the  completion  of  reports  and  studies  to  meet  them, 
including  whether  cost  estimates  have  been  generated.  In  the  past  year,  DOD  has 
initiated  efforts  to  improve  its  visibility  over  its  internal  and  external  reporting 
requirements.  In  March  2011,  the  Secretary  of  Defense  issued  a  memorandum 
requiring  the  Assistant  Secretary  of  Defense  for  Legislative  Affairs  to  enhance  that 
office’s  database  capability  and  further  automate  the  process  of  tracking  external 
reports  to  Congress  and  required  the  Office  of  the  Director  of  Administration  and 
Management  to  track  internal  reports.  The  Office  of  Legislative  Affairs  enhanced  its 
Congressional  Hearings  and  Reporting  Requirements  Tracking  System  by  adding  a 
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data  field  to  record  the  estimated  costs  of  reports  or  studies  generated  by  DOD 
components.  To  gain  greater  visibility  of  internal  reporting  requirements,  the  Office  of 
the  Director  of  Administration  and  Management  tasked  Washington  Headquarters 
Services  with  developing  a  repository  for  tracking  internal  reports.  According  to  a 
Washington  Headquarters  Services  official,  the  repository  has  a  field  to  capture 
whether  an  internal  report  or  study  has  a  cost  estimate.  Currently,  the  repository  is  in 
development  and  has  been  populated  with  reports  that  date  back  3  years. 


DOD  has  not  evaluated  and  currently  does  not  plan  to  evaluate  the  usefulness  of  its 
efforts  to  estimate  the  costs  of  selected  reports  and  studies  as  one  of  the  means  for 
achieving  the  Secretary  of  Defense’s  intended  purpose  of  increasing  the 
transparency  of  costs,  reducing  or  eliminating  reporting  requirements,  and  instilling  a 
culture  of  cost  consciousness  across  DOD.  According  to  GAO’s  Standards  for 
Internal  Control  in  the  Federal  Government ,8  activities  such  as  assessments  need  to 
be  established  to  monitor  performance,  and  managers  need  to  compare  actual 
performance  against  targets  and  then  analyze  any  significant  differences  so  that 
appropriate  action  can  be  taken  to  ensure  that  organizational  goals  are  met. 
However,  under  current  guidance,  no  requirement  exists  for  DOD  to  evaluate  the 
impact  of  its  cost  estimating  efforts,  such  as  whether  these  efforts  have  prompted 
internal  or  external  decision  makers  to  consider  cost  as  a  factor  in  deciding  whether 
to  establish  a  reporting  requirement  for  DOD.  Without  evaluating  the  impact  of  its 
efforts,  including  seeking  the  views  of  internal  and  external  decision  makers,  DOD 
does  not  have  the  information  it  needs  to  assess  whether  the  time  and  effort  it  is 
investing  to  develop  cost  estimates  is  having  the  desired  effect  of  achieving  greater 
transparency,  reducing  reporting  requirements,  and  raising  cost  consciousness,  as 
intended  by  the  Secretary. 


We  are  making  a  recommendation  to  the  Secretary  of  Defense  to  take  steps  to 
evaluate  DOD’s  effort  to  estimate  costs  to  determine  whether  that  effort  is  having  the 
desired  effect  of  achieving  greater  transparency,  reducing  or  eliminating  reporting 
requirements,  and  raising  cost  awareness.  Based  on  the  results  of  this  evaluation,  if 
DOD  plans  to  continue  the  effort  to  estimate  costs  for  selected  reports  and  studies, 
we  recommend  that  the  Secretary  modify  the  current  guidance  or  otherwise  take 
steps  to  improve  DOD’s  cost  estimating  approach.  In  commenting  on  a  draft  of  our 
report,  DOD  partially  concurred  with  our  recommendations,  except  it  disagreed  with 
one  of  the  steps  we  recommended  to  improve  its  cost  estimating  approach.  Our 
comments  on  specific  points  made  in  DOD’s  letter  begin  on  page  14  and  DOD’s 
comments  are  reprinted  in  enclosure  V.  The  department  also  provided  technical 
comments  on  our  draft  report,  which  we  incorporated  as  appropriate. 


8  GAO,  Standards  for  Internal  Control  in  the  Federal  Government,  GAO/AIM D-00-21 .3.1  (Washington,  D.C.: 
November  1999). 
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Background 


In  September  2010,  the  CAPE  office,  in  conjunction  with  the  military  services, 
established  the  Cost  Guidance  Group  to  implement  the  Secretary  of  Defense’s 
requirement  to  estimate  the  cost  of  preparing  DOD  reports  and  studies  and  publish 
that  cost  on  the  front  cover.  This  group,  consisting  of  representatives  from  the  Office 
of  the  Director  of  Administration  and  Management,  the  Joint  Staff,  and  the  military 
departments,  among  others,  developed  the  guidance  and  tool  that  the  Secretary  of 
Defense  directed  DOD  and  its  components  to  begin  using  by  February  1 , 2011. 
Figure  1  provides  a  timeline  of  key  events  related  to  the  CAPE  office’s 
implementation  of  the  Secretary’s  requirement. 


Figure  1:  Timeline  of  the  Development  and  Implementation  of  DOD’s  Cost  Estimating  Tool 


September  30,  2010:  The  Director  of  the  Cost 
Assessment  and  Program  Evaluation  (CAPE)  office 
establishes  the  Cost  Guidance  Group  to  develop 
guidance  and  tool  to  estimate  costs 


August  9.  2010:  Secretary  of  Defense 
gives  a  speech  on  the  DOD  efficiency 
initiatives,  which  include  tracking  and 
publishing  the  costs  of  reports  and  studies 

1 


November  2010:  Cost 
estimating  tool  made 
available  for  beta  testing 
and  guidance  on  use  of 
the  tool  released 


1 


December  27,  2010: 

Secretary  of  Defense  issues  a 
memorandum  mandating  the 
use  of  the  cost  estimating  tool 
for  selected  reports  and 
studies  by  February  1,  2011 


1 


1 


1 


1 


1 


I 


May  2010:  Secretary  of 
Defense  directs  DOD  to 
review  how  it  is  staffed, 
organized,  and  operated. 


August  16. 2010:  Secretary  of  October  25, 2010:  December  1, 2010: 

Defense  issues  a  memorandum  Scheduled  start  Cost  guidance  and 

directing  a  series  of  fourth  track  date  for  support  tool  are  made 

initiatives  designed  to  reduce  development  of  available  on  CAPE 

duplication,  overhead,  excess.  the  cost  estimating  website  to  all  DOD 

and  instill  a  culture  of  cost  tool  and  website.  employees, 

consciousness. 


February  1,2011: 

Cost  estimating  tool 
required  to  be  used 
to  generate  cost 
estimates  for 
selected  reports 
and  studies. 


Source  GAO  analysis  of  DOD  information 


The  tool,  also  referred  to  as  a  calculator,  is  web  based  and  maintained  on  an 
unclassified  DOD  website  (see  enc.  Ill  for  a  snapshot  of  the  tool).  DOD  entities  use 
the  tool  for  training  purposes  as  well  as  to  generate  cost  estimates  that  will  be 
published  on  reports  and  studies.  The  corresponding  guidance  for  the  tool  is  also 
maintained  on  the  website  and  identifies  the  types  of  reports  and  studies  that  require 
a  cost  estimate,  instructs  individuals  on  how  to  access  and  utilize  the  tool,  and 
identifies  manpower  and  nonlabor  costs  that  could  be  considered  for  inclusion  in  the 
cost  estimate,  among  other  things.  The  guidance  and  the  tool  are  updated 
incrementally  with  both  minor  and  major  changes. 


Guidance  on  using  the  tool  identifies  10  types  of  reports  or  studies  that  require  a 
cost  estimate.  Specifically,  a  cost  estimate  is  required  if  the  report  or  study  is 

•  to  be  provided  to  the  Secretary  of  Defense, 

•  to  be  provided  to  the  Deputy  Secretary  of  Defense, 

•  to  be  provided  to  Congress, 

•  to  be  included  in  the  Defense  Technical  Information  Center  holdings, 

•  executed  by  DOD  contractors, 
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•  executed  by  federally  funded  research  and  development  centers, 

•  executed  by  a  board  created  by  DOD, 

•  executed  by  a  board  created  by  law, 

•  executed  by  a  commission  created  by  DOD,  or 

•  executed  by  a  commission  created  by  law. 


In  addition,  although  it  is  not  required,  guidance  on  using  the  tool  recommends  that  a 
cost  estimate  be  generated  for  all  reports  and  studies. 

DOD  Captures  Certain  Costs  for  Selected  Reports  and  Studies,  but  Its 
Approach  Does  Not  Fully  Reflect  Relevant  Cost  Estimating  Best  Practices  and 
Cost  Accounting  Standards 

DOD  is  estimating  and  publishing  approximate  costs  for  selected  internally  and 
externally  required  reports,  but  in  some  cases,  its  approach  is  not  fully  consistent 
with  relevant  cost  estimating  best  practices  and  cost  accounting  standards. 

DOD  Estimates  and  Publishes  Certain  Marginal  Costs  as  Directed  by  the  Secretary 
of  Defense 


DOD  entities  use  the  cost  estimating  tool  to  capture  marginal  costs — costs  for 
activities  that  would  not  otherwise  have  been  performed  had  the  report  or  study  not 
been  developed — associated  with  preparing  and  completing  a  report  or  study.  Costs 
to  be  captured  consist  of  (1 )  manpower  costs,  such  as  the  prorated  salaries  based 
on  time  spent  by  military  personnel  and  DOD  civilians  to  develop  and  publish  a 
report  or  study,9  and  (2)  nonlabor  costs,  such  as  costs  for  travel,  contractor-related 
services,  and  printing.  Once  the  estimate  is  generated,  individuals  include  the  dollar 
amount  of  the  estimate  on  the  front  cover  of  the  corresponding  report  or  study. 
Specifically,  the  language  on  the  front  cover  states  that  “Preparation  of  this 

report/study  cost  the  Department  of  Defense  a  total  of  approximately  $ _ in 

Fiscal  Year _ ”  and  includes  the  date  that  the  cost  estimate  was  generated  and 

a  unique  reference  identification  number  for  the  report  or  study.  CAPE  officials 
responsible  for  developing  the  tool  stated  that  the  Secretary  of  Defense  approved 
this  approach  for  estimating  and  publishing  the  costs  of  reports  and  studies  and 
emphasized  that  the  Secretary’s  intent  was  to  develop  an  easy  to  use, 
nonburdensome  approach  for  estimating  costs.  As  a  result,  the  tool  was  not 
designed  with  the  level  of  precision  of  a  more  formal  cost  estimating  process. 


POP’s  Approach  for  Estimating  Costs  Is  Not  Fully  Consistent  with  Relevant  Best 
Practices  and  Cost  Accounting  Standards 


We  evaluated  DOD’s  approach  for  estimating  costs  against  some  of  the  best 
practices  contained  in  GAO’s  Cost  Estimating  and  Assessment  Guide,  which  states 


9  The  tool  automatically  calculates  salary  costs  for  military  personnel  based  on  composite  rates  and  civilian 
personnel  based  on  the  General  Schedule,  neither  of  which  includes  all  costs  associated  with  manpower. 


8 


GAO-1 2-480R  Defense  Management 


that  high-quality  and  reliable  cost  estimates  should  be  comprehensive,  well 
documented,  and  accurate.  In  addition,  we  compared  DOD’s  approach  with 
Statement  of  Federal  Financial  Accounting  Standards  No.  4  and  relevant  portions  of 
DOD’s  Financial  Management  Regulation,  which  state  that  appropriate  procedures 
and  practices  should  be  established  to  enable,  among  other  things,  the  interpretation 
and  communication  of  cost  information.  We  found  that  DOD’s  approach  in  using  the 
tool  to  estimate  costs  is  not  fully  consistent  with  relevant  best  practices  and  cost 
accounting  standards,  as  discussed  below. 

Comprehensive 

While  GAO’s  Cost  Estimating  and  Assessment  Guide  states  that  cost  estimates 
should  include  all  costs,  it  allows  flexibility  for  the  estimates  to  exclude  costs  where 
information  is  limited.10  In  such  cases,  the  decisions  on  what  costs  are  to  be 
included  or  excluded  should  be  clearly  defined  and  documented  in  the  ground  rules 
and  assumptions  used  to  develop  the  cost  estimate.11  Consistent  with  this  guide, 
DOD’s  guidance  on  using  the  tool  defines  and  documents  decisions  to  include  or 
exclude  certain  manpower  costs  and  to  include  all  nonlabor  costs.  For  example,  it 
identifies  specific  manpower  costs  that  are  excluded  from  the  calculation,  such  as 
costs  associated  with  health  care  and  training.  Additionally,  the  guidance  provides 
broad  examples  of  some  types  of  activities  to  consider  in  estimating  costs,  but  states 
that  individuals  should  use  their  best  judgment  in  determining  what  costs  to  include. 
In  calculating  manpower  costs,  the  guidance  indicates  that  if  individuals  developing 
a  report  or  study  collected  data  or  took  other  actions  that  they  would  have  done  even 
if  the  report  or  study  was  not  required,  the  costs  associated  with  these  activities 
should  not  be  included  in  the  cost  estimate.  For  example,  the  guidance  states 
activities  that  are  part  of  an  individual’s  normal  job  activities — such  as  attending 
meetings;  coordinating  reports,  studies,  or  proposals;  and  completing  security 
reviews  of  documents — should  not  be  included  in  the  manpower  calculation.  On  the 
other  hand,  if  a  study  or  report  required  activities  that  would  not  have  otherwise  been 
undertaken,  costs  associated  with  these  activities  should  be  included  in  the  estimate 
of  manpower  costs.  The  guidance  also  provides  examples  of  nonlabor  activities 
individuals  could  consider  in  developing  the  cost  estimate,  such  as  costs  associated 
with  contractors,  printing,  software  or  hardware  purchases,  lease  or  rental  fees  for 
space,  and  travel. 

Ultimately,  the  guidance  leaves  it  up  to  the  individuals  generating  the  cost  estimate 
to  decide  which  types  of  activities  to  include  or  exclude  in  the  cost  estimate.  As  a 
result,  in  our  review  of  nine  reports  that  DOD  prepared  for  Congress,  we  found 
inconsistencies  in  the  types  of  activities  individuals  decided  to  include  when 
developing  their  cost  estimates  (see  enc.  IV  for  a  description  of  these  reports,  which 
ranged  in  estimated  costs  from  about  $3,000  to  over  $2.3  million).  For  example,  for 
six  of  the  nine  reports  we  reviewed,  we  found  that  individuals  included  in  their 
manpower  calculations  the  time  they  spent  coordinating  the  report  for  review  and 


10  Similarly,  SFFAS  No.  4  permits  flexibility  when  measuring  and  reporting  costs  for  special  purpose  cost  studies. 

11  Grounds  rules  represent  a  common  set  of  agreed-on  estimating  standards  that  provide  guidance  and  minimize 
conflicts  in  definitions.  Assumptions  represent  a  set  of  judgments  about  past,  present,  or  future  conditions 
postulated  as  true  in  the  absence  of  positive  proof. 
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approval  and  the  time  that  their  supervisors  spent  finalizing  it.  For  the  other  reports 
we  reviewed,  costs  for  this  activity  were  not  included.  One  of  the  officials  that 
decided  to  include  time  spent  on  coordinating  the  report  in  the  cost  estimate  stated 
that  DOD’s  guidance  was  unclear  as  to  whether  coordination  was  limited  to  the 
review  and  approval  of  a  report.  Therefore,  given  the  significant  amount  time  he 
spent  coordinating  the  development  of  the  report  (e.g.,  time  spent  clarifying 
guidance  or  coordinating  inputs),  he  decided  to  include  those  costs  in  the  cost 
estimate.  We  also  found  instances  where  costs  for  nonlabor  activities  that  DOD’s 
guidance  states  individuals  could  consider  in  developing  estimates  were  not 
included  even  though  costs  were  incurred.  Specifically,  we  found  that  for  four  of  the 
reports  we  reviewed,  officials  told  us  that  costs  had  been  incurred  for  printing  but 
were  not  included  in  the  cost  estimate.  They  stated  that  they  did  not  include  them 
because  they  believed  their  printing  costs  were  so  small  that  the  level  of  effort 
needed  to  calculate  those  costs  outweighed  the  value  of  including  them  in  the 
estimate.  In  some  cases,  officials  were  not  able  to  readily  retrieve  documentation; 
therefore,  we  were  unable  to  fully  evaluate  the  extent  to  which  they  consistently 
applied  DOD’s  guidance  for  estimating  costs. 

In  addition,  according  to  DOD’s  Financial  Management  Regulation  and  incorporated 
portions  of  Statement  of  Federal  Financial  Accounting  Standards  No.  4,  appropriate 
procedures  and  practices  should  be  established  to  enable,  among  other  things,  the 
interpretation  and  communication  of  cost  information.  While  DOD  has  developed 
boilerplate  language  for  use  when  presenting  the  cost  estimate  on  the  front  cover  of 
a  report  or  study,  the  language  does  not  provide  information  on  what  costs  are 
included  or  excluded.  Specifically,  it  does  not  indicate  that  the  estimate  reflects  only 
certain  marginal  manpower  and  nonlabor  costs.  Rather,  DOD  includes  a  general 
statement  that  says  “Preparation  of  this  report/study  cost  the  Department  of  Defense 

a  total  of  approximately  $ _ in  Fiscal  Year _ .”  Without  further 

explanation,  this  wording  could  be  subject  to  misinterpretation  such  that  recipients  of 
the  reports  or  studies  may  assume  that  all  costs  are  included  in  the  estimate  when 
that  is  not  the  case. 

Well  Documented 

GAO’s  Cost  Estimating  and  Assessment  Guide  states  that  high-quality,  reliable  cost 
estimates  should  be  easily  traceable  to  source  documents.  While  DOD’s  guidance 
on  using  the  tool  states  that  individuals  should  be  prepared  to  explain  how  they 
developed  a  cost  estimate,  it  does  not  include  a  requirement  for  individuals  to  retain 
source  documentation.  According  to  DOD  officials  responsible  for  the  development 
of  the  tool,  the  tool  is  not  a  database  and  therefore  was  never  intended  to  be  used  to 
capture  references  to  supporting  documents.  When  talking  with  officials  responsible 
for  developing  cost  estimates,  we  found  that  for  three  of  the  nine  reports,  they  were 
able  to  easily  retrieve  the  documentation  they  used  to  generate  the  cost  estimate. 
For  example,  some  individuals  were  able  to  provide  us  with  e-mails  containing  data 
on  the  amount  of  time  individuals  spent  working  on  the  report  or  study.  Officials 
responsible  for  three  of  the  other  six  reports  stated  that  they  could  not  easily  provide 
the  information  because  they  had  moved  on  to  other  positions  and  no  longer  had 
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access  to  their  records.  The  remaining  officials  stated  that  with  time,  they  might  be 
able  to  find  and  provide  the  source  documentation. 

Accurate 

GAO’s  Cost  Estimating  and  Assessment  Guide  further  states  that  high-quality, 
reliable  cost  estimates  should  have  all  cost  inputs  independently  checked  to  verify 
that  they  are  accurate.  The  guidance  on  using  the  tool  does  not  include  any  process 
or  requirement  for  another  individual  to  independently  verify  the  manpower  and 
nonlabor  costs  used  to  generate  the  cost  estimate.  In  the  absence  of  any  guidance, 
CAPE  and  service  officials  responsible  for  administering  the  tool  stated  that  they  had 
not  been  assigned  responsibility  for  validating  or  vetting  inputs  used  to  develop  the 
cost  estimate.  According  to  these  same  officials,  the  individual  generating  the  cost 
estimate  is  responsible  for  ensuring  its  accuracy.  In  some  cases,  individuals  who 
were  responsible  for  generating  cost  estimates  for  the  nine  reports  we  reviewed 
stated  that  they  included  the  information  used  to  develop  the  cost  estimates  when 
they  submitted  the  reports  to  their  supervisors  for  approval. 

In  the  absence  of  clearly  defined  parameters  as  to  which  activities  should  be 
covered,  full  disclosure  of  the  scope  of  the  estimates,  the  availability  of  source 
documentation,  and  a  means  to  independently  verify  the  estimates,  DOD  does  not 
have  the  information  it  needs  to  readily  replicate  the  estimates  and  assure  itself  that 
the  estimates  are  consistent,  accurate,  and  reliable. 

DOD  Does  Not  Currently  Have  a  Means  to  Ensure  That  Required  Types  of 
Reports  and  Studies  Have  Cost  Estimates,  but  Has  Efforts  Under  Way  to 
Improve  Its  Visibility 


DOD  currently  lacks  a  means  to  ensure  that  organizations  are  developing  cost 
estimates  for  all  required  types  of  reports  and  studies,  but  is  taking  steps  to  enhance 
its  ability  to  monitor  the  preparation  of  reports  and  studies  to  satisfy  reporting 
requirements,  including  those  for  which  a  cost  estimate  is  required  to  be  generated. 
DOD’s  guidance  on  using  the  tool  identified  10  specific  types  of  reports  or  studies 
that  require  a  cost  estimate.  The  guidance  also  states  that  the  DOD  office  preparing 
the  report  or  study  is  responsible  for  ensuring  that  a  cost  estimate,  if  required,  is 
included.  However,  the  guidance  does  not  include  any  process  or  requirement  to 
track  whether  organizations  are  developing  required  cost  estimates. 


According  to  CAPE  officials  who  developed  the  tool,  they  were  not  directed  to 
ensure  that  cost  estimates  have  been  developed  for  all  required  reports  and  studies. 
They  added  that  the  cost  estimating  tool  was  not  designed  to  track  reports  and 
studies  that  require  a  cost  estimate.  For  example,  officials  stated  that  while  the  tool 
stores  information  each  time  a  user  generates  a  cost  estimate,  they  cannot 
distinguish  between  real  estimates  and  those  that  were  developed  while  individuals 
were  using  the  tool  for  training  purposes.  Therefore,  when  we  requested  a  list  of  all 
the  reports  and  studies  for  which  individuals  had  prepared  cost  estimates,  they  were 
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unable  to  provide  us  such  a  list.  CAPE  officials  also  noted  that  the  magnitude  of 
DOD’s  reporting  requirements  makes  it  challenging  to  identify  the  universe  of  these 
requirements  and  to  track  the  completion  of  reports  and  studies  to  meet  them, 
including  whether  cost  estimates  have  been  generated. 


In  the  past  year,  DOD  has  taken  steps  to  improve  the  department’s  visibility  over  its 
internal  and  external  reporting  requirements.  For  example,  in  March  2011,  the 
Secretary  of  Defense  issued  a  memorandum  requiring  the  Assistant  Secretary  of 
Defense  for  Legislative  Affairs  to  enhance  that  office’s  database  capability  and 
further  automate  the  process  of  tracking  external  reports  to  Congress,  and  required 
the  Office  of  the  Director  of  Administration  and  Management  to  track  internal 
reports.12  The  Office  of  Legislative  Affairs  enhanced  its  Congressional  Hearings  and 
Reporting  Requirements  Tracking  System13  by  adding  a  data  field  to  record  the 
estimated  costs  of  reports  or  studies  generated  by  DOD  components.  To  gain 
greater  visibility  of  internal  reporting  requirements,  the  Office  of  the  Director  of 
Administration  and  Management  tasked  Washington  Headquarters  Services  with 
developing  a  repository  for  tracking  internal  reports.  According  to  a  Washington 
Headquarters  Services  official,  the  repository  has  a  field  to  capture  whether  an 
internal  report  or  study  has  a  cost  estimate.  Currently,  the  repository  is  in 
development  and  has  been  populated  with  reports  that  date  back  3  years.  However, 
officials  stated  that  this  only  captures  reporting  requirements  involving  multiple  DOD 
components  and  does  not  reflect  all  of  DOD’s  internal  reporting  requirements.  The 
official  noted  that  Washington  Headquarters  Services  is  planning  to  send  out  a  data 
call  to  all  of  DOD  in  an  effort  to  capture  the  universe  of  internal  reporting 
requirements. 

DOD  Lacks  a  Plan  to  Evaluate  the  Impact  of  Its  Efforts  to  Estimate  Costs 


DOD  has  not  evaluated  and  currently  does  not  plan  to  evaluate  the  usefulness  of  its 
efforts  to  estimate  the  costs  of  selected  reports  and  studies  as  one  of  the  means  for 
achieving  the  Secretary  of  Defense’s  intended  purpose  of  increasing  the 
transparency  of  costs,  reducing  or  eliminating  reporting  requirements,  and  instilling  a 
culture  of  cost  consciousness  across  DOD.  According  to  GAO’s  Standards  for 
Internal  Control  in  the  Federal  Government,  activities  such  as  assessments  need  to 
be  established  to  monitor  performance  and  managers  need  to  compare  actual 
performance  against  targets  and  then  analyze  any  significant  differences  so  that 
appropriate  action  can  be  taken  to  ensure  that  organizational  goals  are  met. 
However,  under  current  guidance,  no  requirement  exists  for  DOD  to  evaluate  the 
impact  of  its  cost  estimating  efforts,  such  as  whether  it  has  prompted  internal  or 
external  decision  makers  to  consider  cost  as  a  factor  in  deciding  whether  to 
establish  a  reporting  requirement  for  DOD.  In  addition,  CAPE  officials  responsible  for 
developing  and  administering  the  tool  stated  they  have  not  been  otherwise  directed 
to  take  steps  nor  have  they  taken  steps  to  conduct  any  such  evaluation,  including 


12  Secretary  of  Defense  Memorandum,  Track  Four  Efficiency  Initiatives  Decisions  (Mar.  14,  201 1 ). 

13  The  Congressional  Hearings  and  Reporting  Requirements  Tracking  System  is  DOD’s  official  system  used  to 
track  congressional  processes. 


12 


GAO-1 2-480R  Defense  Management 


reaching  out  to  internal  and  external  decision  makers  to  obtain  their  views.  CAPE 
officials  stated  that  anecdotally  they  have  heard  some  officials  remark  that  they 
believe  the  tool  has  increased  the  transparency  of  costs  and  improved  cost 
awareness.  However,  most  of  the  DOD  officials  who  generated  cost  estimates  for 
the  nine  reports  we  reviewed  stated  that  they  had  not  sought  or  received  any 
feedback  from  internal  or  external  decision  makers  about  the  cost  estimates 
published  on  the  front  covers  of  the  reports  or  studies.  Without  evaluating  the  impact 
of  its  efforts,  including  seeking  the  views  of  internal  and  external  decision  makers, 
DOD  does  not  have  the  information  it  needs  to  assess  whether  the  time  and  effort  it 
is  investing  to  develop  cost  estimates  is  having  the  desired  effect  of  achieving 
greater  transparency,  reducing  reporting  requirements,  and  raising  cost 
consciousness,  as  intended  by  the  Secretary. 

Conclusions 

When  the  Secretary  of  Defense  directed  DOD  to  track  the  cost  of  preparing  reports 
and  studies,  his  intention  was  to  generate  an  approximate  cost  that  could  raise 
awareness  of  the  costs  associated  with  the  level  of  effort  DOD  expends  in 
generating  reports  or  studies.  In  response  to  the  Secretary’s  directions,  officials  from 
the  CAPE  office,  in  conjunction  with  officials  from  other  DOD  offices,  expended  time 
and  effort  to  develop  an  approach  to  estimate  the  cost  of  reports  and  studies.  DOD 
components  are  also  expending  time  and  effort  to  use  the  tool  to  develop  cost 
estimates  for  selected  reports  and  studies.  Because  of  this  investment,  it  is 
important  that  DOD  officials  have  an  understanding  of  the  impact  publishing  a  cost 
estimate  is  having  on  internal  and  external  decision  makers.  Further,  there  are 
actions  that  DOD  can  take  to  improve  its  approach  so  that  it  better  aligns  with  key 
characteristics  of  cost  estimating  best  practices — actions  that  would  help  DOD 
ensure  that  it  has  the  information  it  needs  to  readily  replicate  the  estimate  and 
assure  itself  and  the  recipients  of  the  reports  or  studies  that  the  estimates  are  clearly 
defined  and  consistent,  accurate,  and  reliable.  Without  first  evaluating  the  impact  its 
cost  estimating  approach  is  having,  DOD  officials  will  not  be  positioned  to  determine 
whether  the  resources  invested  in  generating  and  publishing  the  approximate  costs 
are  having  the  desired  effect  or  whether  they  should  take  steps  to  improve  their  cost 
estimating  approach. 


Recommendations  for  Executive  Action 

To  determine  whether  its  effort  to  estimate  costs  is  having  the  desired  effect  of 
achieving  greater  transparency,  reducing  or  eliminating  reporting  requirements,  and 
raising  cost  awareness,  we  recommend  that  the  Secretary  of  Defense  take  steps  to 
evaluate  the  effort,  including  reaching  out  to  internal  and  external  decision  makers  to 
obtain  their  views  on  how  it  has  affected  their  decisions  to  establish  internal  and 
external  reporting  requirements  and  whether  they  have  a  common  understanding  of 
what  types  of  costs  are  being  reported. 

Based  on  the  results  of  this  evaluation,  if  DOD  plans  to  continue  the  effort  to 
estimate  costs  for  selected  reports  and  studies,  we  recommend  that  the  Secretary 
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modify  the  current  guidance  or  otherwise  take  steps  to  improve  DOD’s  cost 
estimating  approach,  including 

•  clarifying  those  activities  that  should  be  included  in  the  estimate  as  they  are 
incurred,  including  time  spent  at  every  level  to  review  and  approve  the  report 
or  study; 

•  requiring  that  the  language  on  the  front  cover  of  reports  and  studies  make 
clear  that  the  estimate  is  based  on  certain  marginal  manpower  and  nonlabor 
costs; 

•  requiring  that  source  documentation  used  to  develop  the  cost  estimate  is 
retained  and  easily  accessible  for  review  purposes;  and 

•  establishing  and  implementing  a  verification  process  to  provide  reasonable 
assurance  of  consistency  and  completeness  of  cost  inputs  used  to  develop 
the  cost  estimate. 

Agency  Comments  and  Our  Evaluation 

DOD  provided  written  comments  on  a  draft  of  this  report.  The  full  text  of  DOD’s 
written  comments  is  reprinted  in  enclosure  V.  The  department  also  offered  technical 
comments,  which  we  incorporated  as  appropriate. 

Overall,  DOD  stated  that  it  largely  agreed  with  GAO’s  recommendation  to  evaluate 
the  effectiveness  of  estimating  and  publishing  the  costs  of  DOD  studies  and  reports 
and  that  the  department  remains  committed  to  improving  the  existing  cost  estimating 
guidance,  methods,  and  tools  associated  with  this  policy.  However,  DOD  expressed 
concern  that  we  had  inaccurately  characterized  the  objectives  of  its  cost  estimating 
policy.  For  example,  DOD  believed  that  the  title  of  our  review — Department  of 
Defense’s  (DOD)  Reporting  Cost  Data  Collection — suggested  that  the  department 
was  collecting  cost  data  on  reports.  DOD  also  stated  that  our  draft  report 
inaccurately  stated  that  an  objective  of  the  current  policy  is  to  reduce  or  eliminate 
reporting  requirements.  According  to  DOD,  the  department’s  effort  to  reduce  or 
eliminate  reporting  requirements  is  a  separate  endeavor  entailing  the  formal 
submission  of  legislation  to  Congress,  and  is  inappropriately  linked  in  our  draft  report 
to  the  initiative  to  estimate  and  publish  the  costs  of  DOD  studies  and  reports. 

We  disagree  with  DOD’s  view  that  we  have  inappropriately  linked  the  initiative  to 
estimate  and  publish  costs  to  the  objective  of  eliminating  and  reducing  reporting 
requirements.  We  recognize  that  DOD  has  undertaken  a  specific  effort  to  review  all 
reports  stemming  from  both  external  and  internal  reporting  requirements,  and  have 
clarified  our  report  accordingly.  However,  in  our  review  of  various  DOD  documents, 
including  prepared  remarks  made  by  the  Secretary  and  a  press  release  from  his 
office,  it  is  clear  that  the  review  cited  by  DOD  was  one  of  several  actions,  which 
included  the  effort  to  estimate  and  publish  costs,  that  were  intended  to  address  the 
volume  of  reporting  requirements.  We,  therefore,  modified  the  report  to  provide 
additional  contextual  information  from  these  sources  to  amplify  the  Secretary’s 
intent.  In  the  introductory  paragraphs  of  the  report,  we  now  include  information 
contained  in  the  Secretary  of  Defense’s  August  9,  2010,  speech  announcing  the 
efficiency  initiative  as  well  as  the  accompanying  press  release  further  describing  his 
remarks.  Specifically,  we  note  the  following:  In  his  August  9,  2010  speech 
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announcing  the  overall  efficiency  initiative,  the  Secretary  of  Defense  stated  that  the 
department  is  “awash  in  taskings  for  reports  and  studies”  and  directed  several 
specific  actions  that  according  to  the  press  release  accompanying  the 
announcement,  were  intended  to  “combat  the  enormous  amounts  of  taskings  for 
reports  and  studies.”14  In  his  remarks,  the  Secretary  noted  that  there  is  little  basis  to 
determine  whether  the  value  gained  is  worth  the  considerable  time  and  resources 
expended  to  generate  reports  and  studies.  With  respect  to  specific  actions,  the 
Secretary  directed  DOD  and  its  components  to  track  the  approximate  cost  of 
preparing  DOD  reports  and  studies  and  publish  the  cost  on  the  front  cover  of  each 
report  or  study.  He  also  called  for  a  comprehensive  review  of  all  oversight  reports 
and  use  of  the  results  to  reduce  the  volume  of  reports  and  studies  generated 
internally  while  engaging  Congress  on  ways  to  meet  needs  while  working  together  to 
reduce  the  number  of  reports.  With  regard  to  the  title  of  our  review,  this  title  was 
used  for  administrative  purposes,  including  its  use  in  our  letter  to  notify  DOD  of  our 
work.  We  note  that  in  several  places  throughout  the  draft  report,  we  describe  DOD’s 
effort  as  a  cost  estimating  approach  rather  than  an  effort  to  collect  cost  data. 


With  regard  to  our  recommendation  that  DOD  evaluate  its  effort  to  estimate  the 
costs  of  reports  and  studies,  including  reaching  out  to  internal  and  external  decision 
makers  to  obtain  their  views  on  how  it  has  affected  their  decisions  to  establish 
internal  and  external  reporting  requirements  and  whether  they  have  a  common 
understanding  of  what  types  of  costs  are  being  reported,  DOD  partially  concurred. 
DOD  agreed  that  there  could  be  value  gained  in  evaluating  its  efforts  but  stated  that 
any  such  evaluation  must  avoid  linking  the  efforts  of  this  policy  to  objectives  that  are 
unrelated  to  its  original  intent.  Specifically,  DOD  stated  that  its  policy  was  not 
designed  to  reduce  or  eliminate  reporting  requirements,  which  it  characterized  as  a 
separate  effort.  In  addition,  DOD  stated  that  its  policy  was  not  designed  to  apply  cost 
estimating  best  practices  from  GAO’s  Cost  Estimating  and  Assessment  Guide.  It 
noted  that  DOD’s  guidance  and  methodology  for  estimating  costs  differ  from  criteria 
used  to  estimate  costs  for  other  DOD  decision  making.  It  further  stated  that  DOD 
would  not  support  a  change  to  the  rigor  associated  with  estimating  the  costs  of  DOD 
studies  and  reports  unless  it  is  guaranteed  that  the  marginal  value  gained  by 
increasing  the  level  of  accuracy  outweighs  the  additional  efforts  required  to  produce 
more  accurate  cost  estimates. 

With  regard  to  engaging  with  internal  and  external  decision  makers  to  better 
understand  the  effectiveness  of  the  current  cost  estimating  policy,  DOD  stated  that 
this  process  had  already  begun.  Specifically,  DOD  noted  that  members  of  the 
department  have  communicated  internally  with  DOD  employees  and  externally  with 
non-DOD  employees  to  understand  ways  in  which  the  value  gained  by  the  policy 
could  be  improved.  In  its  comments,  DOD  stated  that  it  had  received  evidence  that 
the  value  gained  by  the  policy  is  significant  and  cited  specific  examples  to  illustrate 
its  point.  For  example,  DOD  noted  that  contact  had  occurred  between  the  Chairman 
of  the  House  Armed  Services  Committee  and  the  Secretary  of  Defense  regarding 
the  costs  associated  with  a  particular  DOD  report,  which  led  to  agreement  that  the 


14  Remarks  as  delivered  by  former  Secretary  of  Defense  Robert  M.  Gates,  The  Pentagon,  August  9,  2010,  and 
DOD  News  Release  No.  706-10,  “Sec.  Gates  Announces  Efficiency  Initiatives”  (Aug.  9,  2010). 
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report’s  quality  required  improvement  and  could  be  produced  at  lower  costs.  In 
addition,  DOD  cited  actions  taken  by  DOD  decision  makers  to  expand  the  scope  of 
the  policy  to  include  additional  types  of  reports  or  to  establish  mechanisms  to  ensure 
that  reports  to  Congress  are  not  transmitted  without  a  cost  estimate.  DOD  also  noted 
that  DOD  decision  makers  have  contacted  CAPE  staff  to  commend  the  Secretary’s 
effort  to  raise  cost  awareness  and  improve  transparency  through  the  use  of  the  cost 
estimating  guidance,  methods,  and  tools.  Finally,  DOD  stated  that  it  will  continue  to 
monitor  the  effectiveness  of  the  current  policy  by  engaging  with  DOD  employees, 
senior-level  DOD  staff,  and  individuals  outside  of  DOD. 

We  are  encouraged  that  the  department  agrees  that  there  could  be  value  gained  in 
evaluating  efforts  to  raise  cost  awareness  and  improve  transparency  of  costs  by 
estimating  and  publishing  costs  of  the  DOD  studies  and  reports.  However,  we 
disagree  with  DOD’s  characterization  of  the  underlying  intent  of  the  evaluation  that 
we  recommended.  As  discussed  previously,  the  Secretary  included  the  effort  to  track 
and  publish  costs  for  reports  and  studies  among  efforts  intended  to  address  the 
volume  of  taskings  for  reports  and  studies.  Therefore,  we  believe  that  in  any 
evaluation  of  the  effort,  it  would  be  essential  to  specifically  obtain  the  views  of 
decision  makers  on  the  benefits  of  estimating  report  costs,  including  how  such 
efforts  affected  their  decision  making  on  whether  to  call  for  future  reports  or  studies. 
As  stated  in  our  report  and  as  reflected  in  the  examples  that  DOD  described  in  its 
response,  we  recognize  that  DOD  has  obtained  anecdotal  feedback  from  internal 
and  external  sources  regarding  the  cost  estimating  effort.  Given  the  time  and  effort 
that  DOD  components  are  expending  to  use  the  tool  to  develop  cost  estimates,  we 
continue  to  believe  that  DOD  needs  to  develop  an  evaluation  approach  that  will  give 
it  a  more  complete  picture  of  how  decision  makers  have  been  influenced  by  the  cost 
estimating  policy.  Without  such  information,  it  will  continue  to  be  difficult  for  DOD  to 
assess  the  value  of  the  policy,  including  whether  the  time  and  effort  it  is  expending 
to  estimate  costs  is  a  worthwhile  investment. 

We  further  note  that  our  recommendation  that  DOD  evaluate  its  cost  estimating 
effort  did  not  suggest  that  such  an  evaluation  would  assess  whether  DOD  should 
apply  cost  estimating  best  practices.  Rather,  we  recommended  that  DOD  first 
evaluate  whether  its  effort  is  achieving  its  desired  effect.  Based  on  the  results  of  this 
evaluation,  we  recommended  that  if  DOD  decided  to  continue  the  effort,  it  should 
modify  existing  guidance  or  otherwise  take  steps  to  improve  its  cost  estimating 
approach  so  that  it  better  aligns  with  key  characteristics  of  cost  estimating  best 
practices.  We  note  that  DOD  specifically  states  in  its  written  comments  that  the 
department  intends  to  continue  the  current  policy.  Therefore,  as  discussed  below  in 
our  response  to  DOD’s  specific  comments  regarding  that  part  of  our 
recommendations,  we  continue  to  believe  that  opportunities  exist  for  DOD  to 
improve  its  cost  estimating  approach  to  make  it  more  consistent  with  relevant  best 
practices. 

With  regard  to  our  recommendations  that  DOD  modify  the  current  guidance  or 
otherwise  take  steps  to  improve  DOD’s  cost  estimating  approach,  DOD  partially 
concurred  with  three  of  our  proposed  recommendations  and  did  not  concur  with  the 
fourth  one. 
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•  DOD  partially  concurred  with  our  recommendation  to  clarify  those  activities 
that  should  be  included  in  the  estimate  as  they  are  incurred,  including  time 
spent  at  every  level  to  review  and  approve  the  report  or  study.  DOD  stated 
that  providing  clarity  to  the  overall  guidance  will  continue  to  be  a  priority, 
adding  that  the  direction  to  exclude  coordination  efforts  in  the  cost  estimate  is 
explicitly  documented  in  guidance.  As  we  noted  in  the  draft  report,  we 
recognize  that  DOD’s  guidance  states  that  manpower  costs  should  not  be 
included  when  participation  only  involves  normal  job  activities,  such  as 
attending  meetings  and  coordinating  studies,  reports,  or  proposals.  However, 
we  found  that  some  of  the  individuals  who  generated  cost  estimates  found  the 
guidance  confusing  with  regard  to  whether  to  include  or  exclude  costs 
associated  with  coordination  activities  that  are  beyond  an  individual’s  normal 
job  activity.  As  a  result,  we  found  inconsistencies  in  how  these  individuals 
decided  to  account  for  this  activity  when  generating  their  cost  estimates. 

•  DOD  partially  concurred  with  our  recommendation  requiring  that  source 
documentation  used  to  develop  the  cost  estimate  is  retained  and  easily 
accessible  for  review  purposes.  DOD  stated  that  its  Cost  Guidance  Portal, 
which  includes  the  tool,  has  several  features  to  assist  users  in  obtaining 
source  documentation  of  their  cost  estimates,  including  an  automatic  e-mail 
with  a  detailed  summary  report  of  the  cost  estimate  and  a  hyperlink  to  each 
cost  estimate.  We  recognize  that  these  features  enable  DOD  to  retrieve  the 
information  that  was  input  into  the  cost  estimating  tool.  However,  it  does  not 
include  a  reference  to  the  actual  source  documents.  As  we  discuss  in  the 
report,  some  individuals  responsible  for  generating  the  cost  estimates  that  we 
reviewed  had  difficulty  retrieving  the  source  documentation  that  they  used  to 
develop  the  estimates.  We  continue  to  believe  that  without  specifically 
requiring  individuals  to  retain  source  documentation,  DOD  may  not  have  the 
information  needed  so  that  others  can  understand  how  a  cost  estimate  was 
derived  and  readily  replicate  it. 

•  DOD  did  not  concur  with  our  recommendation  to  establish  and  implement  a 
verification  process  to  provide  reasonable  assurance  of  consistency  and 
completeness  of  cost  inputs  used  to  develop  the  cost  estimate.  The 
department  stated  that  it  currently  has  the  ability  to  pull  detailed  cost 
estimates  for  specific  studies  and  reports,  when  necessary,  and  to  contact 
each  DOD  employee  who  prepares  a  cost  estimate,  which  DOD  believes 
provides  a  reasonable  level  of  accountability.  DOD  also  stated  that 
implementing  a  verification  process  similar  to  the  one  proposed  by  GAO 
would  be  a  costly  endeavor.  For  example,  verifying  the  amount  of  time  spent 
by  each  DOD  employee  preparing  a  study  or  report  would  require  the 
department  to  reduce  time  spent  on  other,  higher-priority  missions.  Rather 
than  implementing  a  costly  verification  process  for  review  of  cost  estimates 
after  they  have  been  submitted,  the  department  will  continue  to  train  users 
prior  to  their  preparation  and  submission  of  cost  estimates.  We  agree  that  the 
department  can  pull  information  for  a  specific  report  or  study  and  contact 
DOD  employees  regarding  estimates.  However,  we  disagree  that  DOD’s 
ability  to  take  these  steps  constitutes  the  same  type  of  accountability  that 
would  be  present  if  DOD  had  a  verification  process  in  place.  We  are  not 
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suggesting  that  such  a  process  would  require  DOD  to  verify  the  time  spent  by 
each  employee  to  prepare  a  report  or  study.  Rather,  we  would  expect  that 
any  verification  effort  would  include  steps  to  review  whether  the  estimate  was 
developed  in  a  manner  consistent  with  DOD’s  guidance.  Such  a  process 
would  provide  a  greater  assurance  of  consistency  and  completeness  of  the 
cost  inputs  used  to  develop  the  cost  estimate.  In  addition,  while  we  agree  that 
training  is  important,  we  note  that  training  is  not  currently  required  before  an 
individual  is  able  to  use  the  tool,  and  individuals  may  therefore  continue  to 
develop  cost  estimates  without  the  benefit  of  any  training. 

•  DOD  partially  concurred  with  our  recommendation  requiring  that  the  language 
on  the  front  cover  of  reports  and  studies  make  clear  that  the  estimate  is 
based  on  certain  marginal  manpower  and  nonlabor  costs.  DOD  stated  that 
while  the  department  is  open  to  modifying  the  phrase  that  is  published  on  the 
front  cover  of  reports  or  studies,  the  objective  should  not  be  to  detail  the 
specific  costs  included  or  excluded  on  the  front  cover  and  that  the  cost  on  the 
front  cover  should  always  represent  the  marginal  costs  borne  by  DOD  in 
preparing  the  study  or  report.  We  agree  that  DOD  does  not  need  to  detail  all 
the  specific  costs  included  or  excluded  in  the  cost  estimate  on  the  front  cover. 
Rather,  DOD  should  modify  the  language  on  the  front  cover  of  reports  and 
studies  to  make  it  clear  that  the  estimate  is  based  on  certain  marginal 
manpower  and  nonlabor  costs. 


We  are  sending  copies  of  this  report  to  appropriate  congressional  committees  and 
the  Secretary  of  Defense.  In  addition,  this  report  is  available  at  no  charge  on  the 
GAO  website  at  http://www.qao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me  at  (202) 
512-9619  or  pickups@qao.gov.  Contact  points  for  our  Offices  of  Congressional 
Relations  and  Public  Affairs  may  be  found  on  the  last  page  of  this  report.  GAO  staff 
who  made  key  contributions  to  this  report  are  listed  in  enclosure  VI. 


Sharon  L.  Pickup 
Director 

Defense  Capabilities  and  Management 


Enclosures  -  6 
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Enclosure  I:  Description  of  the  Four  Tracks  of  the  Secretary  of  Defense’s 
Efficiency  Initiative 


In  May  2010,  the  Secretary  of  Defense  directed  the  Department  of  Defense  (DOD) 
to  undertake  a  departmentwide  initiative  to  assess  how  the  department  is  staffed, 
organized,  and  operated  with  the  goal  of  reducing  excess  overhead  costs  and 
reinvesting  these  savings  into  sustaining  DOD’s  current  force  structure  and 
modernizing  its  weapons  portfolio.  The  Secretary’s  initiative  targeted  both  shorter- 
and  longer-term  improvements  and  set  specific  goals  and  targets  for  achieving  cost 
savings  and  efficiencies.  The  Secretary’s  initiative  was  organized  along  four  tracks — 
each  of  which  had  a  different  focus,  as  shown  in  table  1 . 15 


Table  1:  Description  of  the  Four  Tracks  of  the  Secretary  of  Defense’s  Efficiency  Initiative 


Track 

Purpose 

Track  1 

Military  services  and  other  components:  Tasked  to  find  savings  of  about  $100  billion  in 
overhead  costs  that  are  to  be  reinvested  over  5  years,  starting  with  DOD’s  fiscal  year 

2012  budget. 

T  rack  2 

External  and  internal  suggestions:  Sought  ideas,  suggestions,  and  proposals  for 
efficiencies  from  outside  normal,  official  DOD  channels  (e.g.,  “think  tanks,”  industry, 
external  boards,  and  DOD  employees). 

T rack  3 

Departmentwide  review  of  certain  areas:  Assessed  personnel  and  health  care  policies, 
logistics,  and  acquisition  to  identify  efficiencies. 

T  rack  4 

Specific  issues  identified  by  the  Secretary:  Focused  on  specific  areas  where  DOD  could 
take  immediate  action  to  reduce  inefficiencies  and  overhead,  in  particular,  to  reduce 
headquarters  and  support  bureaucracies  and  to  instill  a  culture  of  cost  consciousness 
and  restraint  in  the  department. 

Source:  DOD. 


Note:  Other  components  include  various  offices  within  the  Office  of  the  Secretary  of  Defense,  the  combatant  commands,  DOD 
agencies,  and  DOD  field  activities. 


15  GAO,  Streamlining  Government:  Key  Practices  from  Select  Efficiency  Initiatives  Should  Be  Shared 
Governmentwide,  GAO-11-908  (Washington,  D.C.:  Sept.  30,  2011). 
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Enclosure  II:  Scope  and  Methodology 

Our  objectives  were  to  evaluate  DOD’s  efforts  to  implement  an  approach  to 
estimating  the  costs  of  selected  reports  and  studies,  including  (1 )  whether  DOD 
entities  are  capturing  and  presenting  costs  in  a  manner  consistent  with  relevant  cost 
estimating  best  practices;  (2)  the  status  of  DOD’s  efforts  to  track  whether 
organizations  are  developing  cost  estimates  for  all  required  types  of  reports  and 
studies;  and  (3)  whether  DOD  has  evaluated  the  usefulness  of  its  efforts  to  estimate 
the  cost  of  reports  and  studies  as  one  of  the  means  for  achieving  the  Secretary’s 
intended  purpose  of  increasing  the  transparency  of  costs,  reducing  or  eliminating 
reporting  requirements,  and  instilling  a  culture  of  cost  consciousness  across  DOD. 


To  determine  whether  DOD  entities  are  capturing  and  presenting  costs  in  a  manner 
consistent  with  relevant  cost  estimating  best  practices,  we  reviewed  the  cost 
estimating  tool  used  by  DOD  entities  to  generate  cost  estimates  and  compared  it  to 
several  best  practices  recognized  in  GAO’s  Cost  Estimating  and  Assessment  Guide 
and  relevant  managerial  cost  accounting  guidance — Statement  of  Federal  Financial 
Accounting  Standards  No.  4  and  relevant  sections  of  DOD’s  Financial  Management 
Regulation.  GAO’s  Cost  Estimating  and  Assessment  Guide  reflects  best  practices 
for  developing  and  managing  cost  estimates  for  capital  programs.  Because  DOD’s 
cost  estimating  tool  covers  noncapital  costs,  we  assessed  the  tool  against  the  best 
practices  in  the  guide  that  we  identified  as  most  applicable  to  these  types  of  cost. 

We  compared  the  tool  to  three  characteristics — comprehensive,  well  documented, 
and  accurate — in  the  guide  to  determine  if  the  estimate  generated  by  the  tool  is  of 
high  quality  and  reliable.  We  scored  each  best  practice  as  not  met  (the  tool  provided 
no  evidence  that  satisfies  any  of  the  criterion),  minimally  met  (the  tool  provided 
evidence  that  satisfies  a  small  portion  of  the  criterion),  partially  met  (the  tool 
provided  evidence  that  satisfies  about  half  of  the  criterion),  substantially  met  (the  tool 
provided  evidence  that  satisfies  a  large  portion  of  the  criterion),  and  met  (the  tool 
provided  complete  evidence  that  satisfies  the  entire  criterion).  We  determined  the 
overall  assessment  by  assigning  an  individual  rating  to  the  best  practice:  not  met  = 

1 ,  minimally  met  =  2,  partially  met  =  3,  substantially  met  =  4,  and  met  =  5.  To 
perform  this  analysis,  we  reviewed  documents  provided  by  the  Cost  Assessment 
and  Program  Evaluation  (CAPE)  office,  including  guidance  on  using  the  tool  and  the 
tool  itself,  and  conducted  interviews  with  DOD  officials  to  corroborate  our 
understanding  of  the  tool  and  its  corresponding  guidance.  In  addition,  we  compared 
the  cost  estimating  tool  with  Statement  of  Federal  Financial  Accounting  Standards 
No.  4  and  relevant  sections  of  DOD’s  Financial  Management  Regulation  to 
determine  the  extent  to  which  the  tool  reflects  relevant  cost  accounting  standards 
and  cost  management  guidance.  All  data  elements  we  assessed  are  sufficiently 
reliable  for  the  purpose  of  assessing  the  reliability  of  DOD’s  cost  estimating  tool. 
Further,  we  had  originally  intended  to  review  a  representative  sample  of  DOD 
reports  and  studies  with  cost  estimates  published  on  the  front  cover.  However,  DOD 
could  not  readily  provide  us  with  a  comprehensive  list  of  all  reports  and  studies  that 
included  cost  estimates  from  which  we  could  select  reports  and  studies  to  review 
because  DOD  could  not  distinguish  between  real  cost  estimates  and  estimates  that 
were  developed  while  individuals  were  using  the  tool  for  training  purposes.  DOD  was 
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able  to  provide  us  information  on  nine  reports  prepared  for  Congress  on  which 
Congress  had  requested  more  information,  and  identify  the  individuals  responsible 
for  generating  the  cost  estimates  for  these  reports.  Therefore,  we  reviewed  these 
nine  reports.  Using  semistructured  interviews,  we  held  discussions  with  the  key 
officials  responsible  for  developing  the  cost  estimates  for  these  reports  to  gain  an 
understanding  of  their  experience  with  using  the  tool  and  corresponding  guidance 
and  their  level  of  confidence  in  the  cost  estimates  generated,  among  other  things.  In 
addition,  we  reviewed  supporting  documentation  for  some  of  the  estimates  as  well 
as  the  snapshot  of  the  data  that  was  entered  into  the  tool.  In  some  cases,  officials 
were  not  able  to  readily  retrieve  documentation;  therefore,  we  were  unable  to  fully 
evaluate  the  extent  to  which  they  consistently  applied  DOD’s  guidance  for  estimating 
costs.  We  were  unable  to  assess  the  reliability  of  the  data  used  to  generate  the  cost 
estimates  that  were  included  on  these  nine  reports,  or  to  generalize  these  findings  to 
all  reports,  but  we  feel  that  the  review  of  these  nine  reports  is  sufficient  for  providing 
context  about  the  process,  and  we  are  making  a  recommendation  about  future  data 
collection  efforts. 


To  describe  DOD’s  efforts  to  determine  whether  organizations  are  developing  cost 
estimates  for  all  required  types  of  reports  and  studies,  we  reviewed  guidance  and 
interviewed  DOD  officials  to  assess  the  nature  of  their  tracking  efforts.  Specifically, 
we  conducted  interviews,  gathered  information  and  documents  regarding  internal 
and  external  report  tracking  mechanisms  from  officials  in  the  CAPE  office;  the  Office 
of  the  Assistant  Secretary  of  Defense  for  Legislative  Affairs;  the  Office  of  the  Director 
of  Administration  and  Management;  Washington  Headquarters  Services;  and  the 
legislative  affairs  offices  of  the  Departments  of  the  Army,  Navy,  and  Air  Force. 


To  determine  whether  DOD  has  evaluated  the  usefulness  of  its  efforts  to  estimate 
the  cost  of  reports  and  studies  as  one  of  the  means  for  achieving  the  Secretary’s 
intended  purpose  of  increasing  the  transparency  of  costs,  reducing  or  eliminating 
reporting  requirements,  and  instilling  a  culture  of  cost  consciousness  across  DOD, 
we  reviewed  relevant  DOD  guidance  and  GAO’s  Standards  for  Internal  Control  in 
the  Federal  Government.  We  also  interviewed  DOD  officials  responsible  for 
administering  and  using  the  tool,  including  the  individuals  who  generated  the  nine 
reports  for  Congress,  to  determine  if  they  had  taken  steps  to  obtain  feedback  from 
internal  and  external  decision  makers  regarding  the  impact  of  estimating  costs  and 
publishing  the  cost  estimate  on  the  front  covers  of  the  reports  and  studies. 


We  conducted  this  performance  audit  between  October  201 1  and  May  2012  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 
evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions  based  on 
our  audit  objectives.  We  believe  that  the  evidence  obtained  provides  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 
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Enclosure  III:  Snapshot  of  Fields  in  DOD’s  Cost  Estimating  Tool 


The  tool  designed  by  DOD  consists  of  fields  with  drop-down  menus  to  allow  users  to 
choose  the  category  of  costs  they  wish  to  input,  among  other  types  of  information. 
Additionally,  the  tool  contains  information  such  as  the  definition  of  what  constitutes  a 
report  or  study  and  a  real-time  calculator.  (See  fig.  2.) 


Figure  2:  Fields  in  DOD’s  Cost  Estimating  Tool 


Report/ Starfy  (ort  tvtwwtr  (7  301 1038,  Pflt  70200) 

Tburvtoy  27  September  7011 

Fu»po*e 

rIE 

Tho  cost  estnate  taut  M  prepared  as  to  rc  studes  are  readied  for  pubkcato".  The  outpost  y  ttts  estnate  6  to  nprove  the  transparency  of  costs  a  sweatee  «th 
rtoo^s  and  studes  soorwri  v  prepared  by  OoO.  Al  f worts  and  studies  sponsored  or  performed  by  DoO.  rebdrg  those  for  soecal  Kcesi  program*  and  mtory 
rteVje'ce  programs.  nxat  proede  a  cost  estnate.  OOHI  wd  provide  separate  gudance  for  HP  prolans.  AJ  figures  shook)  reflect  actual  costs  to  the  greatest  degree 


Identdy  Report/Stady 


1.  Report/ Study  Tde:  • 

2.  Expected  RubtaCon  Date:  •  3.  Reoort/Studs  Hunter 

09/22/2011  Q 

5.  Name  of  Organosols)  Spomomg  the  Reoort/Study:  • 

6.  Name  of  Oganeaton  prepanng  the  Cost  Estimate:  • 

7.  Th*  report/ study  •  r«ured  to  comply  wth  DoO  reguaoon:  • 

7b.  Enter  the  DoO  regubl»n(*)  recured  for  compbnce: 

8.  Tf«  report/study  >  redured  to  comply  wth  Congressonai  drectcn:  < 
8b.  Enter  the  Congressonai  Report  Hunter 

9.  The  report/study  •  redured  by  fcw:  • 

9b.  Enter  the  Statutory  Ms): 


j 

J 


4.  The  Report/Study  was  funded  and  performed  dumg  Fool  Years:  • 

B»  b 


■ 

B 


[preparation  ol  ft 

w  reporVstuOy  costB 

Ye  Department  of  Defense  a  total  of  a 

ppronmatefy  $0  OR 

yj  Save  S  Dtspl 

ay  wsun 

Rrporl/Study  Cot I  Ittatfr  (70011038.  FK  70700) 


10.  let  costs  of  al  non-ttor  actvtes  used  to  prepare  and  complete  the  report/study  (te.,  contracts,  protng,  travel  etc):  • 


IW%drr  77  September  701 1 


4  VMM  AdMlCcetiSK) 


11.  Total  Government  Manpower  Recured  for  Oreroght  of  Report/  Study  Executon  V:  > 


12.  Withe  cost  estnate  data  be  reduced  on  a  recamg  bas»>:  • 

12b.  Enter  the  freouenev  of  the  recvmg  Reoort/Study  estnate  (ie.,  bennol  annual  quarterly,  etc): 


akXalofappramalelYSO 


Source  DOD 


22 


GAO-1 2-480R  Defense  Management 


Enclosure  IV:  Description  of  Nine  Reports  Prepared  for  Congress  That  GAO 
Reviewed 


We  reviewed  information  on  nine  reports  DOD  prepared  for  Congress  for  which  cost 
estimates  had  been  generated  in  201 1  and  interviewed  the  key  officials  responsible 
for  developing  the  cost  estimates.  The  reports  we  reviewed  were  prepared  by 
organizations  within  the  Office  of  the  Secretary  of  Defense,  the  Navy,  and  the  Air 
Force.  Four  of  the  cost  estimates  consisted  solely  of  manpower  costs,  and  the  other 
five  consisted  of  manpower  and  nonlabor  costs.  (See  table  2.) 


Table  2:  Description  of  Nine  DOD  Reports  Prepared  for  Congress  That  GAO  Reviewed 


Organization 
that  prepared 
the  report 

Report 

Date 

Cost  estimate  (dollars  in  thousands) 

Manpower 

Nonlabor 

Total 

Navy 

Integration  of  Current  and  Future 
Unmanned  Payloads  into  Submarines 

January  12, 
2011 

$3 

$0 

$3 

Office  of  the 

Under  Secretary 
of  Defense  for 
Acquisition, 
Technology  and 
Logistics 

Panel  on  Contracting  Integrity  2010 
Report  to  Congress 

January  28, 
2011 

18 

0 

$18 

Navy 

Report  to  Congress  on  Repair  of  Naval 
Vessels  in  Foreign  Shipyards 

February  8, 

2011 

11 

0 

$11 

Office  of  the 

Under  Secretary 
of  Defense  for 
Policy 

Report  on  the  Joint  Use  of  Federal 

Forest  Fire  Fighting  Assets/C-130  Fire 
Fighting  Capabilities 

February  9, 

2011 

51 

120 

$171 

Office  of  the 
Secretary  of 
Defense,  Cost 
Assessment  and 
Program 
Evaluation 

A  Review  of  the  Army's  Modular  Force 
Structure 

February  16, 
2011 

149 

460 

$609 

Air  Force 

Expeditionary  Combat  Support  System 
(ECSS)  Increment  1  Major  Automated 
Information  System  (MAIS)  Critical 
Change  Report 

February  18, 
2011 

1,232 

1,146 

$2,378 

Navy 

Strategic  Plan  for  Homeporting  the 

Littoral  Combat  Ship 

February  24, 
2011 

11 

0 

$11 

Office  of  the 

Under  Secretary 
of  Defense  for 
Personnel  and 
Readiness 

Department  of  Defense  Foreign 

Language  Proficiency  Fiscal  Year  2010 
Report  to  Congress 

March  2011 

78 

23 

$101 

Navy 

Littoral  Combat  Ship  (LCS)  Monthly 

Cost  Report  to  Congress 

March  24, 

2011 

2 

1 

$3 

Source:  DOD. 
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Enclosure  V:  Comments  from  the  Department  of  Defense 


OFFICE  OF  THE  SECRETARY  OF  DEFENSE 
1800  DEFENSE  PENTAGON 
WASHINGTON.  D.C.  20301-1800 


COST  ASSESSMENT  AND  APR  19  2012 

PROGRAM  EVALUATION 

Ms.  Sharon  Pickup 

Director,  Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Ms.  Pickup: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report,  GAO-12— 
480R,  ‘Defense  Management:  Actions  Needed  to  Evaluate  the  Impact  of  Efforts  to  Estimate 
Costs  of  Reports  and  Studies,’  dated  March  21,  2012  (GAO  Code  351661).” 

The  Department  appreciates  the  opportunity  to  review  and  comment  on  this  draft  report. 
The  Department  largely  agrees  with  GAO’s  recommendation  to  evaluate  the  effectiveness  of 
estimating  and  publishing  the  costs  of  DoD  studies  and  reports,  and  the  Department  remains 
committed  to  improving  the  existing  cost  estimating  guidance,  methods,  and  tools  associated 
with  this  policy. 

As  noted  in  the  report,  the  Department  issued  a  plan  in  August  2010  to  instill  a  culture  of 
cost-consciousness,  in  part  by  estimating  the  costs  of  DoD  studies  and  reports  and  publishing  the 
costs  on  the  cover  page  of  the  study  or  report.  The  objectives  of  this  policy  were  to  raise  cost 
awareness  and  transparency  of  costs  throughout  the  Department. 

To  implement  this  plan,  the  Department  established  the  Cost  Guidance  Group,  which 
consists  of  representatives  from  the  military  departments  and  other  DoD  Components.  Within 
weeks  of  assignment,  this  group  successfully  developed  cost  estimating  guidance,  methods,  and 
web-based  tools  to  assist  DoD  employees  in  estimating  costs  of  studies  and  reports.  The 
Department  directed  all  DoD  employees  to  use  the  guidance,  methods,  and  tools  beginning  in 
February  2011,  and  the  Cost  Guidance  Group  has  continually  made  iterative  improvements  to 
maximize  the  effectiveness  of  this  policy. 

In  September  2011,  the  Department  learned  that  the  GAO  announced  a  new  review  titled 
“Department  of  Defense’s  (DoD)  Reporting  Cost  Data  Collection.”  The  Department  welcomed 
this  outside  review  of  its  new  policy  to  estimate  costs  of  studies  and  reports,  but  was  concerned 
that  the  objectives  of  the  policy  were  misinterpreted  since  the  review’s  title  suggested  that  the 
Department  was  collecting  cost  data  on  reports,  which  is  inaccurate. 

Today,  the  Department  remains  concerned  because  the  Draft  GAO  Report  inaccurately 
states  that  an  objective  of  the  current  policy  is  to  “reduce  or  eliminate  reporting  requirements.” 
The  Department’s  effort  to  reduce  or  eliminate  reporting  requirements  is  a  separate  endeavor 
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entailing  the  formal  submission  of  legislation  to  the  Congress,  and  is  inappropriately  linked  in 
the  Draft  GAO  Report  to  the  initiative  to  estimate  and  publish  costs  of  DoD  studies  and  reports. 

The  Department’s  response  to  GAO’s  recommendation  is  attached.  Should  you  have  any 
questions,  please  contact  my  primary  action  officer,  Dr.  Bryan  Shone,  at  703-692-8045  or 
Brvan.Shone@osd.mil. 


Christine  H.  Fox 
Director 


Enclosure 
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GAO  DRAFT  REPORT  DATED  MARCH  21,  2012 
GAO- 1 2-480 R  (GAO  CODE  351661) 

“DEFENSE  MANAGEMENT:  ACTIONS  NEEDED  TO  EVALUATE  THE 
IMPACT  OF  EFFORTS  TO  ESTIMATE  COSTS  OF  REPORTS  AND 

STUDIES” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  To  determine  whether  its  effort  to  estimate  costs  is 
having  the  desired  effect  of  achieving  greater  transparency,  reducing  or 
eliminating  reporting  requirements  and  raising  cost  awareness,  GAO  recommends 
that  the  Secretary  of  Defense  take  steps  to  evaluate  the  effort,  including  reaching 
out  to  internal  and  external  decision  makers  to  obtain  their  views  on  how  it  has 
affected  their  decisions  to  establishing  internal  and  external  reporting  requirements 
and  whether  they  have  a  common  understanding  of  what  types  of  costs  are  being 
reported.  Based  on  the  results  of  this  evaluation,  if  DoD  plans  to  continue  the 
effort  to  estimate  costs  for  selected  reports  and  studies,  we  recommend  that  the 
Secretary  modify  the  current  guidance  or  otherwise  take  steps  to  improve  DoD's 
cost  estimating  approach,  including  to: 

•  clarify  those  activities  that  should  be  included  in  the  estimate  as  they  are 
incurred,  including  time  spent  at  every  level  to  review  and  approve  the 
report  or  study; 

•  require  that  source  documentation  used  to  develop  the  cost  estimate  is 
retained  and  easily  accessible  for  review  purposes; 

•  establish  and  implement  a  verification  process  to  provide  reasonable 
assurance  of  consistency  and  completeness  of  cost  inputs  used  to  develop 
the  cost  estimate;  and, 

•  require  that  the  language  on  the  front  cover  of  reports  and  studies  make 
clear  that  the  estimate  is  based  on  certain  marginal  manpower  and  non-labor 
costs. 

DoD  RESPONSE:  The  Department  of  Defense  (DoD)  partially  concurs  with  the 
recommendation  set  forth  by  the  GAO. 

The  Department  agrees  that  there  could  be  value  gained  in  evaluating  efforts  to 
raise  cost  awareness  and  improve  transparency  of  costs  by  estimating  and 
publishing  costs  of  DoD  studies  and  reports.  However,  any  such  evaluation  must 
avoid  linking  the  efforts  of  this  policy  to  objectives  that  are  unrelated  to  its  original 
intent.  Specifically,  this  policy  was  not  designed  to: 
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•  Reduce  or  eliminate  reporting  requirements.  Reducing  or  eliminating 
reporting  requirements  is  a  separate  DoD  effort  that  is  not  associated  with 
the  policy  to  include  a  cost  estimate  on  the  front  cover  of  DoD  studies  and 
reports. 

•  Apply  cost  estimating  best  practices  from  the  “GAO’s  Cost  Estimating  and 
Assessment  Guide”  as  cited  in  the  Draft  Report.  The  Department’s 
guidance  and  methodologies  for  estimating  costs  for  this  application  differ 
from  criteria  used  to  estimate  costs  for  other  DoD  decision-making.  The 
Department  would  not  support  a  change  to  the  rigor  associated  with 
estimating  costs  of  DoD  studies  and  reports  unless  it  is  guaranteed  that  the 
marginal  value  gained  by  increasing  the  level  of  accuracy  of  outweighs  the 
additional  efforts  required  to  produce  more  accurate  cost  estimates. 

The  Department  partially  concurs  with  GAO’s  recommendation  to  engage  with 
decision  makers  (internal  and  external  to  the  DoD)  to  better  understand  the 
effectiveness  of  the  current  policy.  This  process  has  already  begun.  Since  the 
Secretary  of  Defense  implemented  this  policy  in  December  2010,  members  of  the 
Department  have  communicated  internally  with  DoD  employees  (both  employees 
who  have  used  the  tools  to  generate  cost  estimates  and  senior-level  decision 
makers  who  have  benefited  from  developments  in  cost  awareness  and 
transparency)  as  well  as  externally  with  non-DoD  employees  to  understand  ways  in 
which  the  value  gained  by  the  policy  could  be  improved.  The  Department  has 
received  evidence  that  the  value  gained  by  this  policy  is  significant: 

•  The  Chairman  of  the  Armed  Services  Committee  contacted  the  Secretary  of 
Defense  to  raise  concern  over  costs  associated  with  a  report  submitted  by 
the  DoD  to  Congress  that  was  considered  unjustified  when  compared  to  the 
value  of  the  report.  This  initiated  a  productive  dialogue  between  senior- 
level  DoD  decision  makers  and  Congress,  with  agreement  that  the  report’s 
quality  required  improvement,  and  could  be  produced  at  lower  cost  to  the 
Department. 

•  Senior-level  DoD  decision  makers  have  expanded  the  scope  of  the  policy 
beyond  congressional  and  internal  DoD  reports  and  studies,  mandating  that 
cost  estimates  be  submitted  with  all  studies  and  reports  prepared  within 
their  Component  (in  addition  to  those  mandated  by  the  Secretary  of 
Defense). 

•  Senior-level  DoD  decision  makers  have  established  enforcement 
mechanisms  to  uphold  the  Secretary’s  direction.  For  example,  the  Assistant 
Secretary  of  Defense  for  Legislative  Affairs  directed  her  staff  to  verify  that 
each  draft  report  published  on  or  after  February  1,  201 1  include  a  cost 
estimate  before  clearing  the  report  for  transmittal  to  the  Congress. 

Similarly,  staff  members  in  the  immediate  Office  of  the  Secretary  of 
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Defense  (OSD)  do  not  accept  studies  or  reports  unless  there  is  a  cost 
estimate  included  on  the  front  cover. 

•  Senior-level  DoD  decision  makers  have  personally  contacted  staff  from  the 
OSD’s  Cost  Assessment  and  Program  Evaluation  (CAPE)  office 
commending  the  Secretary’s  efforts  to  raise  cost  awareness  and  improve 
transparency  through  use  of  the  cost  estimating  guidance,  methods,  and 
tools  provided  on  the  Cost  Guidance  Portal. 

The  Department  will  continue  to  monitor  the  effectiveness  of  the  current  policy  by 
engaging  with  DoD  employees,  senior-level  DoD  staff,  and  individuals  outside  of 
the  DoD. 

With  respect  to  the  specific  recommendations  offered  by  the  GAO  to  modify  the 
current  guidance,  the  Department: 

•  Partially  concurs  with  the  recommendation  to  “clarify  those  activities  that 
should  be  included  in  the  estimate  as  they  are  incurred,  including  time  spent 
at  every  level  to  review  and  approve  the  report  or  study.”  Providing  clarity 
to  the  overall  guidance  will  continue  be  a  priority  for  the  Department.  The 
direction  to  exclude  coordination  efforts  in  the  cost  estimate  is  explicitly 
documented  in  the  guidance. 1 

•  Partially  concurs  with  the  recommendation  to  “require  that  source 
documentation  used  to  develop  the  cost  estimate  is  retained  and  easily 
accessible  for  review  purposes.”  The  Cost  Guidance  Portal  includes  several 
features  to  assist  users  in  obtaining  source  documentation  of  their  cost 
estimates.  Immediately  after  completing  a  cost  estimate: 

o  An  automatic  email  with  a  detailed  summary  report  of  the  cost 

estimate  is  sent  to  the  DoD  employee  who  prepared  the  cost  estimate, 
o  A  detailed  summary  report  of  the  cost  estimate  is  available  for  DoD 
employees  to  save  electronically  or  print  for  their  records, 
o  A  unique,  permanent  hyperlink  to  each  cost  estimate  is  provided  for 
DoD  employees  to  save  as  a  bookmark  in  their  web  browser  so  that 
they  can  access  and  recover  their  estimate  in  the  future. 

•  Nonconcurs  with  the  recommendation  to  “establish  and  implement  a 
verification  process  to  provide  reasonable  assurance  of  consistency  and 
completeness  of  cost  inputs  used  to  develop  the  cost  estimate.”  The 
Department  currently  has  the  ability  to  pull  detailed  cost  estimates  for 
specific  studies  and  reports  when  necessary.  As  mentioned  in  the  GAO 


1  GAO’s  draft  report  states  “DoD’s  guidance  was  unclear  as  to  whether  coordination  was  limited  to  the 
review  and  approval  of  a  report,”  (page  1 1).  However,  DoD’s  Studies  and  Reports  Guidance  Document 
states  “Manpower  costs  should  not  be  included  when  participation  only  involves  normal  job  activities  such 
as  attending  meetings,  coordinating  studies/reports/proposals,  completing  security  reviews  of  documents, 
etc.”  (page  7). 
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Draft  Report,  the  Department  has  extracted  a  sample  of  detailed  cost 
estimates  for  selected  studies  and  reports  to  show  to  the  Congress.  The 
Department  also  has  the  ability  to  contact  each  DoD  employee  who 
prepares  a  cost  estimate,  which  provides  a  reasonable  level  of 
accountability.  Implementing  a  verification  process  similar  to  the  one 
proposed  by  the  GAO  would  be  a  costly  endeavor.  For  example,  verifying 
the  amount  of  time  spent  by  each  DoD  employee  preparing  a  study  or  report 
would  require  the  Department  to  reduce  time  spent  on  other,  higher  priority 
missions.  Rather  than  implementing  a  costly  verification  process  for  review 
of  cost  estimates  after  they  have  been  submitted,  the  Department  will 
continue  to  train  users  prior  to  their  preparation  and  submission  of  cost 
estimates. 

•  Partially  concurs  with  the  recommendation  to  “require  that  the  language  on 
the  front  cover  of  reports  and  studies  make  clear  that  the  estimate  is  based 
on  certain  marginal  manpower  and  non-labor  costs.”  While  the  Department 
is  open  to  modifying  the  phrase  that  is  published  on  the  front  cover  of  the 
study  or  report,  the  objective  should  not  be  to  detail  the  specific  costs 
included  or  excluded  on  the  front  cover — this  level  of  detail  is  currently 
provided  on  the  Cost  Guidance  Portal  and  in  the  automatic  email  sent  to  the 
employee  who  submits  the  cost  estimate.  Although  each  study  and  report 
requires  a  unique  level  of  manpower  efforts  and  non-labor  costs,  the  cost 
estimate  presented  on  the  front  cover  should  always  represent  the  marginal 
costs  borne  by  the  DoD  in  preparing  the  study  or  report.  That  is,  the  cost 
estimate  should  include  any  costs  associated  with  activities  the  Department 
would  not  have  encountered  had  the  study  or  reporting  requirement  not 
been  assigned.2 

The  Department  will  continue  to  implement  the  current  policy  with  consideration 
of  these  specific  recommendations  to  improve  the  cost  estimating  guidance, 
methods,  and  tools  associated  with  this  effort. 


2  There  are  some  exceptions.  For  example,  coordination  of  draft  studies  and  reports  is  an  internal  DoD 
activity  that  is  brought  upon  by  the  DoD’s  own  policies.  Therefore,  the  costs  associated  with  coordination 
should  not  be  included  in  the  cost  of  the  study  or  report. 
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